Repository navigation
feat: push to Underlay with the v2 delta protocol - #3690
Merged
Merged
Conversation
Underlay v2 removes negotiate. A push now reads the head's manifest, diffs our records against it by (type, id) and hash, and uploads only upserts and deletes through a push session (POST /push, /records, /deletes, async /commit). - hash: JCS (RFC 8785) written as a string, matching the server; the old sorted-object JSON.stringify put integer-like keys first. canonicalize stays for PubPub's own signatures. - client: readHead (paged manifest, pinned to the head semver), diffAgainstHead, batches sized by the session's line and byte limits, session abandoned on failure, 409 conflict / "No changes detected" told apart, missing files uploaded then a new session, presigned upload for files over 32 MB, metadata sent as a patch of changed fields only, default base URL on www (the apex 301 turns POSTs into GETs). - incremental: resolve cache-hit records by (type, id) and self-heal cache entries whose stored hashes no longer match. - worker: cache persisted on noop too; commit change counts in the push log. - /accounts/me lists orgs as `orgs`, not `accounts`: the org picker and the connection test's org check now work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Underlay protocol v2 removes the negotiate protocol our "push to Underlay" uses. This moves the push to v2's delta push.
How it works now: the push reads the head version's manifest and diffs our records against it by (type, id) and hash. It then opens a push session, uploads only the upserts and deletes, and commits asynchronously.
Changes
Testing: typecheck and biome pass, and the 107 Underlay unit tests pass. client.test.ts was rewritten against a fake Underlay push API. Read-only against staging, our record and schema hashes match the server's for 9,828 records. Not yet run: a real push against staging.
Deploy: don't merge before http://www.underlay.org switches to v2, because this client can't talk to v1. Ship it at the same time as that switch, and pause scheduled pushes in between.